Data compression is desirable in many situations for efficiently storing, providing, maintaining, and/or transferring data. In systems which maintain large data sets, eliminating data redundancy is paramount to efficiency. The redundancy increases storage requirements and has a detrimental impact on read/write performance because caches become less efficient. In environments where millions of large records have to be maintained, this presents a significant challenge. Furthermore, in some situations, it is necessary to compress data incrementally where records are added one by one over the life cycle of the system.
In one case, network management systems are required to collect large amounts of data from devices. When centrally managing home networks, this may be millions of devices. This data may include configuration settings, live performance and fault data, device logs, etc. For example, a service provider may desire to actively manage consumer home devices and backup entire device configuration settings several times a day in order to offer customers a restore function or to handle seamless replacement of devices while preserving user-settings. The data that needs to be stored is often highly redundant across millions of devices. Compressing individual records in isolation is not efficient because this does not take advantage of similarities across data records.
In another case, designated servers of a distributed management system may generate management policy for devices based on templates and distribute the generated policy instructions to enforcement elements. It may not be desirable to distribute information about some common templates the policy is derived from across the system, but rather it may be preferable for enforcement elements to be able to receive a complete set of instructions for each client and compress data incrementally, which provides for a more loosely coupled system design.
Embodiments of the present invention and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures. It should also be appreciated that the figures may not be necessarily drawn to scale.